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ABSTRACT 

Instructional System Technology in recent years has 
been characterized by an increase in individualized instruction and 
the modularization of the curriculum. In traditional systems the 
learners are forced to take blocks of instruction the size of entire 
courses and these are much too large. The courses can now be broken 
down into conceptual subdivisions and these into series of modules. 
Each module has clearly stated behavioral objectives regardless of 
the nature of the intellectual performance prescribed in it. The test 
items of the module are derived from the objectives rather than from 
the instructional materials currently in use to attain them. Ti.e 
instructional materials for a module may be offered on different 
tracks which feature different levels of abstraction thus 
accommodating the different learning characteristics of individual 
learners. Most instructional system developers today are looking 
ahead to a time when good modules will be developed at many 
localities and circulated widely tor use at various institutions. 
There is a growing need for a valid and widely applicable method of 
evaluating modules of instruction. Three basic criteria are relevant 
to the evaluation of a module of instruction; (1) the amount of 
learning that results, (2) the dollar equivalent cost to the 
instructional orga lization of providing the module, and (3) the 
amount of learner r.ime consumed by interaction with the module. An 
example is given comparing three different versions of a module. 
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I, The Individualization and Modularization of Instxuction 

Instructional System Technology in recent years has been character- 
ized by an increase in individualized izistruction and the modularization 
of the curriculum, through the system ap^^roach the benefits of 
individualization have been applied to large masses of students. It 
has been demonstrated that it is not necessary to instruct in large 
groves vMch are lodged together by a mass uniform presentation. Groups 
are new taught as individuals v/ith self-pacing and differing learning 
prescriptions based on the characteristics of the individual learners. 

By breaking doMn the content into modules of small size it has edso 
become a sinnple matter to combine modules in different ways for different 
learners. Modularization has provided the brecucthrough to custom tailor- 
ed curriculums with mudi flexibility in crossing disedplinary lines. 

In traditional systems the learners are forced to take blodcs of 
instruction the size of entire courses and these are much too large. 

The coiuoses can now be broken down into conceptual subdivisions and 
these into series of modules. A module may be as small as a single 
lesson and be based on a single oonc^>t. Some modules may stand ad.^ie 
and be included singly in any individual's curriculum. In other cases 
a series of modules may be linked in a sequence of increaising complexity 
so that prior ones cu:e prerequisites for the next. 

Each module has clearly stated behavioral objectives regardless 
of the nature of the intellectual p^formanoe prescribed in it. The test 
items of the module are derived from the objectives rather than from the 



instructional materials currently in use to attain them. test 
items are used to provide both a pretest and posttest for each module. 

A sufficiently high pretest score permits the learner to skip over 
the module and go on to a subsequent one. The same criterion level 
score is ordinarily required of the learner on the posttest in order 
for the learner to move on to the next module. 

The instructional materials for a module may be offered on different 

a 

tracks which feature different levels of abstraction thus accomodating 
the differing learning characteristics of individual learners. For 
exanple, within a single module one track may consist of a printed 
narrative viiich es^lains the content in the style of a textbook; a 
second track might break the content down into smaller steps and pri^* 
sent it more slowly, pednaags as a slide tape presentation using more 
visuals and featuring frequent stops for student responses on a special 
response sheet. A third track for this module might consist of a linear 
program with carefully stmctured small st^ at a relatively concrete 
level. Which track a learner would pursue might be determined by a 
oonbination of the pretest score and the learning characteristics of the 
individuad. Ihe choice of track could be left vp to the learner vho 
, would be provided with the information necessary to make an appropriate 
decision. 

Most instructional system developers today are looking ahead to a 
time vhen good modules will be developed at many loc al ities and circulated 
widely for use at various institutions. Such an exchange of modules will 
vastly increase the potential curriculum at any individual school. Projects 
are underway vAiich are attenpting to collect and evaluate existing modules 
from different sources and this trend is likely to oontimae and to in- 



crease. Ml Institutions will then have acasaa to xnore resources than do 
the richest ones today. 

II. The l^easurement of Instruction 
A. CRLTBRIA 

There is a growing need for a valid and widely applicable method 

« 

of evaluating nodules of instruction* There is a need for a general 
index of goodness to indicate hew good a module is relative to other 
modules dealing with different content. There is also a need for a method 
of qualitatively raxiking two or more \ersions or editions of the same 
module, i.e. ones v^ch share the same objectives and test. For those 
producing modules it is of much practiced iiiportance to knew whether or 
not various changes or subsequent editions represent improvements, and 
vdiether the local product is si^ierior to a version of a module inported 
from elsebdiere. 

— Thf^ basic criteria are relevant to the evaluation of a module of 
instruction: 1) the amount of learning that results from use of the 
module, 2) the dollar equivalent cost to the instructional organization 
of providing the module of instruction, and 3) the amount of learner time 
consumed interaction with the module. To instruct efficiently is to 
maximize learning ^^le mixiimizing do3.1ar and time costs. All instructional 
organizations try to acocnplish as much change as possible to their 
students with as little resource e^qpenditure as possible. These oonsider- 
•ations are summarized in what might be called the "instructional aooGnplish* 
ment index". 

Instructional Aoccxnplishment « UBARNING 

DSLLAR ob^ k iflABMER 
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The instructional acocnplishment index 63 <{>resses output per unit 

» 

of input; the produced oonmodity is learning, defined as a change in 
behavior; the expended resources are both the system's money and the 
learner's time. Glhese tMO resources are given equal wei^t in the 
index, because it is inportant to give full consideration to the interests 
of the learners and to avoid hiding weak instruction by forcing students 
to expend excessive amounts of their time making iqp for its deficiencies. 

B. UNITS OF MEASURE 

Learning ; Behavioral objectives are specified for each module of 
instruction and the test items are written relative to those objectives. 

Because the test items relate to the objectives, th^ can, and should be 
written prior to the design of the specific instructional activities 
intended to lead the students to the attainment of those objectives. 

Correct responses on posttest items that would not have been answered / 

correctly on the pretest represent changes in behavior that constitute 
learning. Ihus the gain scores across the pre- and posttest can be 

I 

taken as measures of the student's learning. If the test scores are V 

expressed as per cents, then the learning can be es^ressed in "units'' of 
percentage points. However, for use in the I^ index, the measure is 

✓ 

^ slightly altered from simple gain scores as explained in the next section. 

V 

The figure below illustrates the significant features of a measure- 
ment of a student's perfozmanoe on a modules 

arbitrary 

criterion 
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The baseline represents the degree of attairanent of the bdiavioral 
objectives of the module in terms of test scores from 0% to 100%. The 
criterion level is arbitrarily set and is frequently at 90%. Hbr skills 
involving the use of the surgeon’s knife the criterion level would 
ordinarily be higher, while for skill at successfully kicking football 
field goals from distances greater tiian midfield the criterion level 
03 cdinarily be less than 90%. Regardless of vdiere the criterion 
level is set, a posttest score must equal or exceed the criterion level 
in order to be regarded as a* valid posttest. Otherwise the test’iig ex* 
perienoe is sin|)ly counted as another learning activity, and another 
attempt to get a permissible posttest score is subsequently arrang^. 

As the figure en^phasizes, the pscetest score, the posttest score, and 
the range between them are all variables. 

An analysis of the above measure of learning reveals bi«o factors: / 

1) It is desirable to obtain the highest possible posttest score, 

2) It is desirable to maximize the gain score. Once a pretest level 

1 

is ifih<ad the goal for the learner is to score 100% on the ^ 

posttest, i.e. attain the highest percentage of possible gain vdiere 
possible gain is the difference bet.veen 100% and the pretest score, 
s The instructional potential of a module should be capable of producing 

y 

a gain range extending close to the 100% level- deemed ade q ua t e if a 
module were able to bring a zero pretest stude n t only to the 70% 
level and no higher even though by relative standards a gain sooaDe of 
70% would usually be considered quite high. 

The cono^t of maximizing the percentage of gain between the pretest 
score and 100% emxmpasses both the oonoepb of maximizing the posttest 
score and maximizing the absolute value of the gain score. Thus maximizing 
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the percentage o£ gain possible beyond the pretest score enierges as the 
most significant and fundamental oonoept. Ihe notion of desirability for 
a high posttest score is included, and so is the notion of a high absolute 
value on the gain score. But new the placeonent of the gain score range on 
the zero-to-lOO test score scale is taken into account insomuch as, regard- 
less of how wide the gain score range is, its effect is measured in terms 
of hew close to the tc^ of the scale it reaches. 

The mathematical es^resaion of the attained percentage of possible 

c S 

ga5n score beyond the pretest level is £ i x 100, where S« is 
the final or posttest score and is the initial or pretest score. The 
value of this expression is used for the learning in the numerator of the 
index. 

Cost ; The total cost to the instructional organization of providing 

/ 

the instruction can be e3g>ressed in dollars, because the cost of all 
resources e3<pended by the ^'stem, including personnel time, can be converted 
to their dollar equivalents. Cost factors, in addition to materials and 
salaries, might include depreciations, rents, taxes, utilities, amortizations, 
or other direct or indirect costs. Ihe particular oonbination of factors 
to be included would depend on the nature and purpose of the specific 
X instance of measurement. When determining the cost per student to provide 
a module of instruction it is necessary to amortize initial production 
costs across the useful lifetime of those products and to consider the 
number of students vto will actually use than. ^ 

Time ; The learner's time of interi’iction with a module of instruction 
is simply measured in hours and is the total time for pursuing instructional 
activities, testing, being tutored, or otherwise assisted. 
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C. CMiCULATING THE I 
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The nondnal units of the instructional aocxxiplislsnent index are thtis 
"percentage points per dollar-*hour" • The index value is nomally ocnputed 
on the basis of a large saniple of students use a given nodule of 
instructicn. The learning and time factors are the OGnputed means for 
all students using the module. The cost factor is the mean cost per 



student. 

The instructional acocxtplishment index is the resultant learning 
divided by the product of the cost to the instructional system to provide 
the experience and the learner's time oansumed by involveDnent with the 
instruction. Tb v arning measure is essentially leased on changes in 
test scores, and t..ei..,i:ore iiS, the change in scores, is the symbol chosen 
to represent it. AS is equal to ^f ^i x 100 as es^lained in the 

100 - Si 

previous section. The sum of all inten^als of learner instructicxi occurring 
between the pre- and posttest is divided by the number of students and 
this liisan is denoted by T and is e9q>ressed in hours. The sum of all 
relevant costs incurred by tha instructional system in pocoviding the 
module of instruction is divided by number of students^ and that cost 
per student is denoted by C and is expressed in dollars. 

The foxmula for the instructional acocnplishment index is thus 

SS . 

C X T 



D. GRAPHIC ANALYSIS OF TEIE INSTBOCTTONAL AOOOMPLISHMBNT INDEX 
If a module of instruction is to be improved aooording to tlie I^ index, 
then the new version must increase the aS/CT ratio. This can be done 
either by 1} ocramiting further resources in an effort to get a dispropor- 
tionately higher amount of learning, 2) altering the nature of the 
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learning activities vmile maintaining the same net CT value in an effort 

to get more learning, or 3) reducing the net CT resource ir^t v^le 

trying to get a less*‘than-*proportional decrease in learning. Figure 1 

• • 



graphically illustrates these alternatives. 




o 

ERIC 



FIG. 1 OF INCREASES IN 1»E 4S/CT Pmo. 



If the initial version of a module oonsunes the resources indicated 
by position 1 on the CT axis and results in a level of learning indicated 
by position 1' on the AS axis, then point UO establishes the straic^t 
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diagonal line on the previous figure. If a second version is then 
doveloped which utilizes more resources as indicated position 2 
on tlie CT axis, reference to point (D) above and hence to position 
2' on the CiS axis gives the breakeven point for resultant inprovement 
in learning (tlie anticipated directly prc^rtional increase) • The 
same value of would result. For a higher 1^ value, the resultant 
learning from the resource increase would have to clinb at sane dis-" 
proportionately higher rate, indicated by the da sh ed curve, so that 
sane higher point (C) would be reached and a level of learning (2”) 
would be attained. A similar argument can be made for reductions in 
the CT resource irput. An effort to provide different activities at 
the same resource level would sixrply be an effort to get more learning 
(point D) through an equally es^ensive but different instructional 
strategy. 

Figure 1 is thus useful for determining the perfarmance required 
of a subsequent version in order to be considered an improvement over 
the existing one. 

Figure 2 illustrates the variations in as the resource ixput 
varies while learning remains constant. In cases \diere it is believed 
possible to reduce the resource expenditure without incurring ax^ de-* 

V 

crease in learning, this figure is useful in predicting the resulting 
higher I. value. For exarrple, if one version of a module results in a 
AS of 90% (lower curve) , for a (CT) resource input of *9 dollar-hours, 
the resultaht 1^ value would be 100. If a new versicxi could be produced 
that would require an e)q)enditure of only .6 dollar*hours, then fion 

I ' 

the graph it can be seen that the I^ vedue Amid rise to 150 provided AS 
did not ch£mge. If the same exanple had involved t ^ distant at the 
100% level (vpper curve) r then I. would have risen from 111 to 167* 
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FIG. 2 IHE lNDB>i: AS A FUMCIION OF EXPENDED BESODRCES WZIIH USAIS1IN6 HELD CSCNSXMir 
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It is easy to see in figure 2 hew the value of increases vdth a 
positive acceleration as resources are conserved and the learning level 
is maintained. This results in a disproportionately higher reward for 
the increasingly efficient use of resources. ^Oie value is very 
sensitive to variations in resources near the lower limit, especially 
within the 0. 0-1.0 dollar^hour range. 

Ill- /plications of the Instructional Acoocplishment Index 
A. OjMPARISai OP VI3BSI0NS FROM DWsmsm SOURCES 
The following exanple involves the use of the 1^ index to OGn^>are 
three different versions of a module. These versions are the products 
of different design teams, and the situation is typical of what might 
be exgected if the same module were developed at three different institutions. 
Once a module is defined in terms of its behavioral objectives and there 
is agireement on what shall constitute a valid test of the attainment of 
those objectives, then the instructional aoocmplishm^t index provides a 
useful method of comparing all completed versions that come forth from 
whatever source. 

Supgose that a biology module is being prepared covering the sprouting 

characteristics of com, and that the objectives have been specified and 

the tests written. Three versions of the module exist, each featuring a 

different set of inst3nx±ional activities intended to insure the learner's 

attainment of the same set of behavioral objectives. These versions 

« 

represent the efforts of three different teams of instructional designers 
each of vbic±i has produced tdiat it believes to be the best approach to the 
t^lsk• 

1. VERSIONS : The first teem of designers decided that th^s 



cmtent could best be taught by using realistic plastic models of com 
sprouts in various stages of grcwth. purchased sudi a set of 

materials from a biological sv^ly house and buir: a module around the 
use of these models* Iheir design utilizes tape xecorded directions for 
the learners wlio are told hew to examine the various models and to re- 
cord their observations on a special respcxise i^eet* At the end of the 
lesson the narrator sums up what the student should have learned. The 
toted developnnent cost for materials and labor to aooemodate 4 students 
is $250 for the ocxnpleted module plus an additional 3^ per student for 
furnished expendable materials. 

The second team of designers discount the value of the models and 
instead use a slide-t£^ program featuring photographs of real com sprouts 
and illustrative liiie drawing done by an experienced Instructional illus- 
trator* A studerit response sheet is used during the program for recording 
responses requested by the program, and a summary in the form of a set 
of written notes is provided to each student for purposes of review study. 
Ihe total development cost for materials and led) is $375 plus an additional 
4^ per student for furnished es^endable materieds* 

The third team of designers s\:^ly each learner with a quantity of 
real com sprouts which have been preserved in a solution and acquired 
from biological suppliers* The learner gets a list of questions vhich 
he is expect&dL to answer by examining the preserved specimens. In 
addition, this team shoots a 15 minute Siper-8 motion picture vhich uses 
time lapse j^togr^hy to shew com grains in the process of sprouting. 

The total development cost of this module is $525 plus an additional 10^ 
per student f^x furnished expendable materials* 

2* TESTING THE VERSIONS < SuppoBB that these three versions of 
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the nodule are tested on three similar groi^ of learners. Each test 
group has 400 students and is randomly drawn fron a pool of 1200 students 
having a pretest noau of 40%. 

The first gzov^, using the plastic models with tcqped directions arid 
a response sheet, tended to waste some of their time in exoessiveXy han- 
dling the models and trying to discriminate their features. This groiqp 
had a mean tinne of 48 minutes and a mean posttest score of 88%. Ihe 
* second group, using the more structured and paced slide-t£^ c^roach with 
response sheet and notes, had a mean time of 42 minutes and a posttest 
mean of 92%. The higher test score was apparently the result of good 
structuring of the basic presentation and the student's use of the notes 
for review. The third groi:^, using the preserved specimens and the film, 
had a mean time of 56 minutes and a posttest mean score of 93%. Ihe 
preserved specim^ were mes^ to use and it was more difficult to see 
their characteristics than when either the plastic models or the good 
line drawings were used. Ihus the students took a longer time with this 
version, yet their test results were the highest of the three gxoips. 

3. DETERMINING THE <3QST PER STUDENT ; The question is hew to 
rank tiiese three instructional acoonplishments. Before £^lying the 
instructional accemplishment index to these three cases, it is first 
necessary to determine what costs are to be oonsi^iered. The production 
costs in ‘':erms of labor and materials have been specified for the three 
versi^:is as $250, $375, and $525. For the sake of this example ^ it 
shall be assumed that in each case si 2 f ficient materials were produced to 
enable a groip of 400 students to be accomoda t ed, and it shall further be 
assumed that the necessary facilities to accomodate them in the intended 
way are available. This might be a Learning Center vhich can provide 
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individualized instruction to as many as 400 students must get 
through a given module within a specified period of time* Ragardless of 
the version of the module being used by a student r in this exanple an 
overhead cost factor of 5^ per student will be included in the oost 
figure for each student stud/ing a module of instruction in the facility* 

The oost factor used for the instructional acocmplishmsnt index ocmputa- 
tions will tlius consist of the amortized per student production oost, 
the per student oost of furnished e)qpendable materials, and the over-* 
head factor for facility use* 

The amortized per student production oost is not meaningful if 
it is siKply amortized over the interval of the test if the materi als 
actually last longer than the test interval or would be suitable for use 
with groii^s larger than the test gxoi:^* For example, a oostly set of slides 
might be used by the test groi^ and suffer almost no depreciation either 
through wear or through the outdating of their content* To divide the 
production oost by the number of test subjects would result in a much 
higher per student oos actor than would be the case if the oost were 
divided by the total nu^^ser of students who could use the slides during 
their potential useful life* Similarly, the oost of a videotape broadcast 
to the test gzov^ would have a much higher oost-per-student factor for 
the test grovp alone than if the production costs were divided by the vast 
numbers of potential users might use the videotape if it were bxoadr 
cast as part of an extention course* Tbe quadity of an instructional 
accxDmplishment must be based in part on the potential real oost per 
student rather than on the oost per student only during the test* ^us, 
while the test interval is all that need be oonsidezed in order to detezmina 
valid means for the test scores and the learner time oonsunsd by the 
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xnodule, if either the useful life or the capacity of the materials 

ft 

exceeds the time limit or nunber of students in the test gxoqp respectively 
than, tlie cost per student must be determined ori the basis of the projected 
longivity of the materials or the magnitude of their audience in normal 
xmtest use. 

Production costs must be broken doMn intp two categories, the initial 
design and production costs that are incurred only onoe and the costs of 
reproduction of multiple copies of the materials. It is only the forn»r 
costs which axe amortized over the useful lifetime of the module. Ihe 
latter costs involving the reproduction of multiple copies of the materials 
are amortized over the useful lifetime of the material. A version of a 
module may remain in use a long time during which the students wear out 
numerous copies of its materials. 

It is difficult to anticipate the useful life of a module, because 
that life can be texminated by a number of unforeseen factors such as 
the introduction of new knowledge, new curriculxsa revision plans, or the 

% 

introduction of a superior version. IStevertheless, an estimate does need 
to be made so that the initial develcproent costs can be amortized realis- 
tically. 

Ihe anticipated physical life of materials subjected to repeated 
* 

use by students is sansRohat easier to estimate. Extrapolations can be 
made on the basis of physical deterioratiois mecisured during the test 
interval. If the test interval is not of sufficient length to provide 
useful deterioration data, then it may be useful to conduct independent 
destruction tests on the materials and search out other test results on 
the physical life of such materials. ^2!hese costs can be leduoed to a 
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fixed increment per utilization \ 4 hid 1 amortizes the cost of the materials 
over that number of utilizations determined to be their mean physical 
life. 

Let us DCM return to our exaiiple. We sliall assume in this case that 
the content is sufficiently stable so that no updating of the content 
because of nem kncwledge is prdsable in the relevant future. Ihereforer 
for purposes of oonparing the three versi^m of this module about com 
sprouting it will be presumed that each version would have a life of 
three years, a somewhat arbitrary interval beyond vMch a si:$)erior version 
would have been foimd to r^laoe the initial one. If the test grovqp of 
,400 students is taken to be the number would ^pically stu^ this 
module during a semester, then over a three year period about 3,000 
students would use the module assuming summer session loads were about 
one-third as heavy as during regular semesters. Therefore, initial 
development costs, excluding multiple copies of materials, are to be 
amortized over 3,000 students in order to compute the development cost 
per student. 

The expendable materials must be treated in two categories: 1) those 
materials furnished to each student and not retrieved for use by others, 
and 2) those materials v^ch are used r^seatedly by many students and 
v^ch wear out over a period of time mod have to be replaced. The former 
type consists mostly of single use documents such as worksheets or 
response sheets. Ttiese costs for the three modules have alrea^ been 
specified as 3<-, 4^, and 10^ per student. Ihe following t^le shows the 
breakdown of costs for initial development (amo^zed over 3,000 students) 
and the cost of the multiple copies of reusable materials (to be amortized 
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over their mean pl\ysicaX lifetixnes) 



Initial Dev. 
Plus Copies for 
400 Students 


Initial Dev. 
Only 


Cost Per 
Student 
(3,000 
Students) 


fnitial Copxes 
(cost for 400 
students) 


Cost Per 
Student 
(reusable 
materials) 


$250 


$200 


6.7« 


$ 50 


1.90 


$375 


$225 


7.5« 


$150 


6.20 


$525 


$400 


13.3« 


$125 


9.40 



TMSLE I: AMOmzmCMS OF INITIAL DBWiOFMEIir AND MDUTIPI£ OOPY CSOSIS. 



In the above table column a:ie is the sum of columns two and four; 
oolisnn three is column two divided by 3,000 (the nusober of students v«ho 
would use the module during the three year lifespan of the version) • 

Column five is oorputed by estimating on the basis of esqorienoe the 
mean physical lifetime of the reusable materials (t^os, slides, filrs, etc.) 
in terms of the nunber of individxaal utilizations they can stand (based 
in this example cn 400 per semester and 400/3 per sanroer session) • This 
number of students is then divided into the cost of those multiple copies 
vhich is listed in column fo^ar. 

An escamination of column two reveals a substantially higher develop- 
xnent cost for the third version. While the first two involved the develop- 
ment of tapes and slides, the third version involved the developoent of a 
time lapse motion picture. In general, it is more costly both in time and 
mon^ to design and produce good iCDtion pictures, and this is reflected. 

Ttie differences in column five reflect both the differences in cost 
for multiple copies as reflected in coluoan four and the very different 






18- 



rates at v^ch the different t?^s of materials wear out. l^e first 
version had a low initial multiple copy cost because it involved only 
tape recordings. In addition, sinple tape recordings laist a relatively 
long time ooarpared to either synchronized slide-tape materials or motion 
picture films. Because the tapes woald last almost as long as the pro- 
jected life of the versiui (three years) , the nurber .of students vho 
would use these initial eopies is just sli^tly under the 3,000 total 
students who would use the version over its lifetime. The actual estimate 
is 2,600 students for which the $50 cost amortizes to 1.9^ per student. 

There is more to go wrong with a slide-tape program both because 
of the addition of slides and because the autonatic slide advance pulses 
on thie tape increase the probability that the tape will not ccmtinue to 
function properly. Hence, the physical lifetime of the initial materials 
for version two is projected to be slightly less than that of the sinple ^ 

tapes in version one. It is estimated that the slide-tape materials 

t 

would last only through the first 2,400 students for a cost per student 
of 6.2«. V 

Version three involves the multiple use of copies of Stper-8 motion 
picture film vhich cost a total of $125 to provide. However, film does 

✓ 

not stand as many plays as do slides or tapes; prints beocms scratdied, 
sprocket holes tear, and splices beocroe frequent as the breaks are repair* 
ed. It is therefore estimated that the fiim prints will last only three 
semesters plus the intervening surtmer session thus serving 1,333 students. ^ 

The initial multiple copy cost of $125 for the prints thus amortizes over 
1,333 students to yeild a cost per student of 9.4^. 

The following table summarizes the various costs for each of the 
three versions with all costs esqpressed in teirins of the cost per student 
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per interaction vdth the nodule. 





Version I 


Version II 


Version III 


Use or Instructional Facilities 




O? 


■ 'S'.O*' “ ■ 


Initial Develojpment Amortized 
Over Qlhree Years 


6.7<: 


7.5« 


13.3<: 


Single Use Materials 


3.0C 


4.0« 


10.0« 


Multiple U&Q Materials Amortized 
Over Their Expected Pl^sical Life 


1.9<: 


6.2<! 


9.4« 


COST PER STUDENT: 


16.6« 


22.7^ 

il 


37.7<> 

— 



TABLE II: SUMMARY OF ODSTS PER STUDEMT FOR TUB OHREB VERSIGilS 



4. DET1H»CENING Tlffl LEARNING itp ) : Before a CGRpitation of the 
indices can be made for tiie three versions, it is necessary to 
the learning factors (Q^S) for use in the numerator of each 1^^ (see sections 



II-B and II-C) • The following table gives the dS learxiing factor for 



each version based on the posttest results presented in section III-Ar>2. 






i VERSION I 


— vmmm ’ 


Version III 


AS a Sf - ^ „ 


88 - 40 X 100 


92 - 40 X 100 


93 - 40 X 100 


V S . 


0 

1 

o 


l6o - 40 


B'6 ■-■ 40 




B 80.0 


= 86.7 


= 88.3 



TABLE III: lEARfSraS FACTORS FOR TOE THREE VERSIONS FDR USB IN 
OOMPUTATION OF THE I^ INDICES. 



These values represent the attained percentage of those objectives 

x^naining to be attained beyond those alreac^ attained at the time of the 

« 

e 
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pretest* 



5. TIM E pm COST ESOPRESSED FOR USE IN THE OOmjTPaHOtS i Time 
is to be expressed in lx>urs. Fran section ZXI-A-*2 the contperative tests of 
the versions yielded neon times of 48, 42, and 56 minutes respectively 
vihich equal .800 hr., .700 hr., and .934 hr. 

Ohe costs per student e}q>ressed in dollars, fron the bottcin of Table II, 
are .166 dollars, .227 dollars, and .37? dollars. 

6. CXMPTOKCION OF THE TOREE INDEX VNJJESi 



VEBSim I 



Ia " AS 
C X T 



8Q.0 

.166 X .8o0 



VERSION II 

I*« AS 

51Tf 



« 86.7 

.221 X .W 



VERSION III 




AS 

^ITT 



88.3 



« 603 » 546 = 248 

7. INTERPBETmON AND ANALYSIS OF THE TOSEE I^ Vl^yUES: According 
to the results above the first module represented the best instructional 
acocnplishment. It must be noted hOMever, that it did well because of its 
low cost. Ihis first module failed to yield a posttest mean of 90% during 
the test period; if a criterion level of 90% were being employed, then a 
large number of students used this version would not be attaining tiie 
criterion level by the time of their first attenpt at a posttest. Forcing 
all students scoring below 90% on the posttest to continue studying tiiie ^ 

module until th^ could score at that level would result in the time 
factor increasing sufficiently to reduos the I^^ value well beXa# that of 
the second version. 



Both the second and third versions resulted in sufficient learning to 
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yield irean posttest scores above 90%, but the excessive dollar and time 
costs associated with version three resulted in its much lower index* 

Although version three resulted in a two per cent higher postte:^c score, 
the vast difference in the index for versions two and three sugg^t 
that the c^roach used in version three should probably be in 

favor of slight but hopefully effective refinements of versicxis one or 
two. 

It is obvious that great eoanomies can result in an extremely low 
denoninator, especially in the dollar cost factor. For exarrple, a 
versicxi might consist of xnatericds limited to only mimeogjre^ahed pages 
of print. The cost would be only a few cents per student, and with even 
modest learning on the part of the students, the value could become 
very high. Ihis tends to hempen because the first points scored on tests 
usually come easily while it beoomes increasingly difficult to attain an / 

increasingly higher score. Chec^ and less effective materieds may result 
in a disproportionately high score level on the posttest, to a 
sufficient level to provide enough numerator value in the expression ^ 

so titiat the value can beocxne high, and yet Z\S will still not reach 
a satisfactory level of learning. Ihe performance of version one in the 

✓ 

^ previous exanple is an instance of this phenomenon. Ihe solittion is 

it 

either to iiipose a crj/^erion level on the mean posttest score usixig first- 
attempt scores, or to lorce all students to make the criteria level score 
prior to ocnputing the value sp that their extra time spent in re- < 

^cling is reflected in an increased time factor in the dencminator. 

B. CX3MPARES0N OF SUOCESSIVB EDm0^lS OF A MCXXXCfi 

An instructional organization vhich is producing and utilizing 
modules of instruction should constantly be collecting data on the per- 
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fozxnanoe of its nodules and tuzning out new editions of them on the 
basis of that feedback. Ihe use of the index pexndts a oamparison 
of each new edition witli that of the older ones to see if the desired 
iirprovement has resulted. 

c. amEsmjG for funds 

Much instructional develo^mont is funded by government agencies, 
foisndations, or other granting institutions, and ooRisetition for the 
funds is frequently strong. An agency may be confronted with similar 
proposals from two or more sources each claindng to be both ocnpetent 
and efficient in the design of instruction. Ihe agency could sicciply 
grant each ocnpetitor a small sum to produce their version of a test 
module for \Aiidci the granting agency would specify the behavioral objec- 
tives. A reasonable time limit could be set, and the agency could sv^)er- ^ 

vise an independent test of the resultant versions. Ihe resulting 

1 

values could be used as an indication of the competence' of the oonpetitors 

with subsequent substantial grants going to those vdio demonstrated their ^ 

si:perior abilities. In this way, for the esqenditure of a few months time 

t 

and a few thousand >v)liars, an agency could have a far more fair and 
objective way of determining to whom it should give its money. The cost 
of such a test is small ocnpared to the many instructional development 
grants vhich range into the hundreds of thousands of dollars. 

D. VTURIABILTIY OP THE VMU& FOR A GEVEN VERSION v 

The I^ value associated with a version of a module is not an intrinsic 
prcpeirty of that version, but is instead associated with a particular 
situation ixwolving its use. From one situation to another there may be 
changes in the nature of the target population which will result in 
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significantly different values and (T) values. There may also be 
different factors entering into a deterxidnation of the cost per student. • 
For exan^le, one organization may develop a version and have to amortize 
the development costs; later it may sell sone copies to another institution 
vdiidh would ocxipute its cost per student on the basis of an amortization 

I 

of the purchase prioe. Ihe develcponent and purchase costs may be very 
different, i Thus the of a version must be reexmputed for the situation 
of its current use or concern, but siKih necessary data as might remain 

relevant can be carried forward and included in any new calculation of the 

\ 

value. 
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